Reproducible research and the substantive context.

In a recent editorial in Biostatistics, Peng (2009) introduced a system for making data and computer code from papers in the journal publicly available so that other researchers can reproduce the statistical analyses in the paper. The editors explain in a short accompanying note (Diggle and Zeger, 2009) that ‘we wish to encourage the practice of making research published in the journal reproducible by others’. Peng’s proposal follows an earlier commentary by Peng and others (2006) that had caught the attention of the influential journal Annals of Internal Medicine (cf. Laine and others, 2007). I agree that it is commendable to be able to reproduce the statistical calculations that a group of authors has done, and Peng has put much care into developing a framework that should allow precise documentation of data and computer code. Hopefully, the focus on reproducibility will enhance the level of documentation of original research data and of the actual calculations that were done, a documentation which may sometimes leave quite a bit to be desired. Peng’s editorial also actively encourages others to conduct “alternative analyses.” This could sometimes be constructive if the purpose was to develop new methodology, but then the term “reproducible” is a misnomer. From the context, it seems that the purpose of these alternative analyses concerns the substantive issues, and my intention with this note is to point out that this is less innocent than it may look at first glance. We have severe caricatures in our tradition that should make us careful. There is usually much more to “research” than the statistical analysis, and there is much more to the statistical analysis than the correctness of calculations. A reasonable choice of statistical model (here “correct” is not a meaningful descriptor) requires deep insight into the substantive context, but it is quite unclear how a satisfactory role for the substantive context is assured in the new system. We should avoid encouraging the already far too widespread occurrence of poorly informed reanalyses of old data sets about which the statistician knows almost nothing, a culture that apparently bothers editors of (other!) statistical journals surprisingly little. In my view, there is no guarantee that the “reproduction” that comes out of this culture will benefit science.